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(54) Helicopter virtual image display system incorporating structural outlines. 

(57) A virtual image display system provides video 
displays based upon virtual images of the exter- 
nal world having synchronized structural out- 
lines superimposed on the video displays to a 
pilot operating an aircraft such as a helicopter 
in non-visual flight conditions. The virtual im- 
age display system includes a virtual imaging 
subsystem for generating virtual images of the 
external world, a video display subsystem for 
generating video images based upon the virtual 
images and for displaying the video images for 
the pilot's viewing, a sensing means for provid- 
ing signals corresponding to the spatial loca- 
tion and perspective of the video display 
subsystem, a map comprising structural out- 
lines corresponding to structural members 
forming the canopy structure of the helicopter, 
and a computer subsystem providing electronic 
interfacing between the elements of the virtual 
image display system, for synchronizing the 
orientation of the virtual imaging subsystem 
with the video display subsystem, and for pro- 
cessing the virtual image signals to provide 
signals to the video display subsystem to gener- 
ate video displays. The computer utilizes the 
helmet position signals to define the relative 
position and orientation of the video display 
subsystem in the cockpit, and utilizes such 
definition to reconstruct structural outlines 
from the map that are synchronized to the 
perspective of the video images. The 
synchronized structural images are superim- 
posed upon the video display. 
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Related Application 

The present application is related to U.S. patent 
application Serial No. 07/160,222, filed February 24, 
1988, entitled CONTACT-ANALOG HEADUP DIS- 
PLAY METHOD AND APPARATUS, and U.S. patent 
application Serial No. 07/160,059, filed February 24, 
1988, entitled AIRCRAFT HELMET POINTING AN- 
GLE DISPLAY SYMBOLOGY. 

Technical Field 

The present invention relates to aircraft display 
systems, and more particularly, to a virtual image dis- 
play system that provides video displays based upon 
virtual images of the external world in combination 
with corresponding superimposed cockpit structural 
outlines to helmet mounted display subsystems for 
aircraft, particularly helicopters, operations during 
non-visual flight conditions such as night and/or ad- 
verse weather flight operations. 

Background of the Invention 

Future generation aircraft (including helicopters) 
now in planning and/or development phases (as well 
as many present generation aircraft) are complex sys- 
tem comprised of a large number of interrelated, com- 
plex subsystems such as the airframe, powerplant, 
flight controls, avionics, navigation equipment, arma- 
ment, etc. Such subsystems generate significant 
amounts of status data, much of which must be fre- 
quently monitored by the pilot for the safe and/or effi- 
cient operation and/or pilotage of the aircraft. A con- 
siderable portion of pilot workload in these future gen- 
eration aircraft will be devoted to monitoring the status 
of the aircraft subsystems during flight operations via 
reference to the generated status data. 

In addition, mission requirements for such future 
generation aircraft may involve a greater percentage 
of high pilot workload flight operations such as nape- 
of-the-earth (NOE), adverse weather, and/or night fly- 
ing. Such high pilot workload flight operations require 
the pilot to maintain a continual spatial awareness of 
aircraft orientation and/or location with respect to the 
external world and a situational awareness of objects 
of interest in the external world vis-a-vis the aircraft in 
addition to continual monitoring of the status of aircraft 
subsystems. 

It will be appreciated that the task of monitoring 
the status of the various aircraft subsystems may con- 
flict with the tasks of maintaining continual spatial and 
situational awareness of the external world. To mon- 
itor the status of aircraft subsystems, the pilot may 
have to divert his attention from the observation of the 
external world outside the cockpit to reference gener- 
ated status data. Such diversions may lead to losses, 
in varying degrees, of spatial and/or situational 



awareness of the external world, which, in turn, may 
lead to less than optimal flight conditions, especially 
during high pilot workload flight operations. 

Current aircraft design methodology strives to op- 

5 timize the interrelationship between the functional 
task of monitoring aircraft status information and the 
functional tasks of maintaining continual spatial and 
situational awareness of the external world vis-a-vis 
the aircraft. Such design methodology seeks systems 

10 and methods that allow vital aircraft status information 
to be accessible to the pilot in such a manner that 
there is no interference with the continual spatial and 
situational awareness functions being performed by 
the pilot. In addition, such aircraft status information 

15 should be presented in a manner consonant with the 
spatial orientation and perspective of the pilot to pre- 
clude any decoupling among the various functional 
tasks. Such decoupling may lead to increased pilot 
workload (to maintain a viable frame of reference) 

20 and/or to pilot disorientation. 

Electro-optical systems have been developed to 
provide aircraft status information to the pilot to facil- 
itate simultaneous accomplishment of both the status 
monitoring functions and the spatial and situational 

25 awareness functions. These systems generate sym- 
bolic and digital status information images that corre- 
spond to the aircraft status information generated by 
the various aircraft subsystems and superimpose 
such symbolic status information images into the pi- 

30 lot's field of vision. The images are typically intro- 
duced into the pilot's field of vision by means of colli- 
mated light rays so that the symbolic images appear 
to be at optical infinity with respect to the pilot's visual 
system. 

35 Thus, the pilot views the external world outside of 

the cockpit at infinity and simultaneously sees sym- 
bolic and digital images at infinity. The superim posi- 
tion of two sets of images, i.e., the external world and 
electronically generated symbolic and digital status 

40 information images, enable the pilot to simultaneously 
maintain awareness of the status of the aircraft, the 
spatial orientation of the aircraft with respect to the ex- 
ternal world, and a situational awareness of the exter- 
nal world vis-a-vis the aircraft. 

45 Exemplary prior art electro-optical systems utiliz- 

ing collimated light rays to generate symbolic images 
include head-up display (HUD) subsystems and hel- 
met mounted display (HMD) subsystems. Represen- 
tative examples of HUD and HMD subsystems are il- 

50 lustrated in U.S. Patent Nos. 4,446,480, 4,439,775, 
4,439,157, 4,305,057, 4,269,476, and 3,923,370. 
While such electro-optical systems have contributed 
significantly to the optimization of the interrelationship 
between the functional task of monitoring aircraft sta- 

55 tus information and the functional tasks of maintaining 
continual spatial and situational awareness of the ex- 
ternal world vis-a-vis the aircraft during both visual 
and non-visual flight conditions, a problem has been 
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identified that interferes with the functional tasks of 
maintaining spatial and situational awareness of the 
external world vis-a-vis the aircraft 

During visual flight conditions, the pilot utilizes a 
cognitive mechanism that allows the pilot safely and s 
efficiently maintain continual spatial and situational 
awareness of the external world vis-a-vis the aircraft. 
More specifically, the pilot, either consciously or sub- 
liminally, utilizes the canopy structure of the aircraft as 
a frame of reference to orientate his visual view of the 1 o 
external world vis-a-vis the aircraft This cognitive 
mechanism precludes pilot disorientation during vis- 
ual flight conditions inasmuch as the canopy structure 
provides an effective reference for maintaining spatial 
and situational awareness of the external world. A 15 
useful analogy would be the automobile driver who 
utilizes the hood and front bumpers of his automobile 
to maintain a continual spatial and situational aware- 
ness of the external world, i.e., the relationship of his 
automobile to the road, other traffic, pedestrians, etc. 20 

During non-visual flight conditions such as night 
and/or adverse weather flight operations, a pilot can- 
not generally rely on the cognitive mechanism descri- 
bed in the preceding paragraphs. Many time the non- 
visual flight conditions are such that the pilot cannot 25 
visually perceive the canopy structure and/or the ex- 
ternal world, and therefore cannot effectuate a refer- 
ential relationship between the canopy structure and 
the external world to facilitate maintenance of spatial 
and situational awareness of the external world. 30 
Moreover, to an increasing degree, pilots operating an 
aircraft in non-visual flight conditions utilize virtual im- 
ages of the external world, which are coupled into the 
pilot's field of view by means of the aforede scribed 
electro-optical systems, for pilotage of the aircraft 35 

Such virtual images may be generated by various 
sensor systems such as low-light television systems, 
infrared imaging systems, etc. A common character- 
istic of such virtual image generating systems is that 
the frame of reference of such generated virtual im- 40 
ages of the external world is totally divorced from the 
canopy structure. For example, a virtual image gener- 
ating system is typically located in the nose section or 
the forward belly section of an aircraft so as to gener- 
ate virtual images of the aircraft's projected flight path. 45 
Such virtual images, especially where the aircraft is 
engaging in flight maneuvers other than straight and 
level flight, cannot be easily correlated to the canopy 
structure of the aircraft, which provides an essential 
underpinning for the utilization of the aforedescribed 50 
cognitive mechanism during flight operations. 

As a result, pilots utilizing virtual images to pilot 
aircraft in non-visual flight conditions may become 
easily disorientated when engaging in flight maneuv- 
ers other than straight and level flight. Such disorien- 55 
tation adversely affects the safe pilotage of the air- 
craft, especially in high pilot workload flight conditions 
such as NOE flight operations. A need exists to pro- 



vide pilots utilizing virtual images for non-visual flight 
conditions with an aircraft referent that will allow the 
pilot to utilize the cognitive mechanism that the pilot 
utilizes during visual flight conditions to maintain con- 
tinual spatial and situational awareness of the exter- 
nal world vis-a-vis the aircraft during flight operations 
utilizing virtual images of the external world. 

Summary of the Invention 

A primary object of the present invention is to pro- 
vide a virtual image display system that provides vid- 
eo displays based upon virtual images of the external 
world in combination with superimposed cockpit 
structural outlines to a pilot via a helmet mounted dis- 
play subsystem. 

This object and other objects may be achieved by 
a virtual image display (VID) system according to the 
present invention for use in piloting aircraft, particular- 
ly helicopters, during non-visual flight conditions such 
as night and/or adverse weather flight operations. The 
VID system according to the present invention in- 
cludes a primary processing, interface, and control 
means, a virtual imaging subsystem, a digitized map 
of the aircraft canopy structure, a video display sub- 
system, and a means for sensing the attitude and pos- 
ition of the video display subsystem with respect to a 
predetermined aircraft coordinate system. 

The VID system according to the present inven- 
tion is operative to provide continuous virtual images 
of the external world, to generate continuous video 
displays based upon these virtual images of the exter- 
nal world, and to transmit such video images to the pi- 
lot via the video display subsystem for use in piloting 
the aircraft during non- visual flight conditions such as 
night and/or adverse weather flight operations. The 
VID system is further operative to superimpose a syn- 
chronized portion of the structural outlines comprising 
the digitized map of the canopy structure upon the vid- 
eo images such that the pilot is provided with a cog- 
nitive video display orientated with respect to the hel- 
icopter canopy structure, thereby allowing the pilot to 
reference the video displays representative of the ex- 
ternal world vis-a-vis corresponding aircraft canopy 
structure. 

The VID system of the present invention has par- 
ticular utility for use in helicopters during non-visual 
flight conditions such as night and/or adverse weather 
flight operations. The helicopter includes a movable 
virtual imaging subsystem and a cockpit having a pre- 
determined canopy structure. The virtual imaging 
subsystem is operative to provide continuous virtual 
images of the external world during non-visual flight 
conditions, i.e., those flight conditions wherein the pi- 
lot's visual system is unable to comprehend the exter- 
nal world with sufficient clarity to ensure safe and/or 
efficient operation of the helicopter, e.g., during night 
and/or adverse weather flight operations. To generate 
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virtual images of the external world during night flight 
operations, a forward looking infrared (FLIR) imaging 
subsystem may be utilized to generate continuous, 
high-definition virtual images of the external world 
during night flight operations. Other imaging systems 5 
such low-light television systems, radar systems, or 
other sensors that are capable of generating high- 
definition virtual images of the external world may be 
utilized as the virtual imaging subsystem in the VI D 
system according to the present invention. The virtual 10 
imaging subsystem is slaved to the video image dis- 
play subsystem such that the FOV of the virtual imag- 
ing subsystem is movable with respect to the helicop- 
ter to align the FOV of the FLIR subsystem to coincide 
with the optical axis of the visual system of the pilot 15 
of the helicopter. 

The video image display subsystem of the VI D 
system is operative to position video images of the ex- 
ternal world (derived from the virtual images captured 
by the virtual imaging subsystem) centered with re- 20 
spect to the optical axis of the visual system of the pi- 
lot, i.e., within the field of view of the pilot. The video 
image display subsystem may be a helmet mounted 
display (HMD) subsystem that includes a helmet and 
a visor type screen. The HMD subsystem is operative 25 
to project collimated video images, derived from vir- 
tual images provided by the virtual imaging subsys- 
tem, onto the pilot's visual system to provide a realis- 
tic, high-definition pictorial representation of the exter- 
nal world during non-visual flight conditions, or view- 30 
ing by the pilot's visual system. 

Associated with the helmet of the HMD subsys- 
tem is a sensing means that is operative to define the 
spatial position and angular orientation of the HMD 
subsystem within the cockpit of the helicopter. Posi- 35 
tion/orientation signals developed by the helmet sens- 
ing means are coupled to the primary processing, in- 
terface, and control means which is operative to proc- 
ess such signals to identify the position and angular 
orientation (perspective) of HMD subsystem in rela- 40 
tion to the cockpit. 

An imaginary reference point, designated as the 
"design eyepoint", is defined in the cockpit of the hel- 
icopter. The design eyepoint is a design parameter 
that defines the degree of visibility (unimpaired visual 45 
field of view of the external world) provided to the pilot 
of the helicopter based upon the structural configura- 
tion of the canopy structure. 

The design eyepoint defines an origin of a coor- 
dinate system for the VID system of the present inven- so 
tion. To facilitate the identification and description of 
the spatial location and perspective of the HMD sub- 
system by means of the primary processing, interface, 
and control means, a polar coordinate system is util- 
ized to define the spatial location and perspective of 55 
the HMD subsystem and to generate the digitized 
map of the canopy structure. Identifying the spatial lo- 
cation and perspective of the HMD subsystem is gen- 



erally simplified by use of the polar coordinate system. 
As a general rule, once a pilot is properly positioned 
in the cockpit seat, the majority of head movements 
(which directly correspond to HMD subsystem move- 
ments) naturally made by the pilot during typical flight 
operations consist of head rotations about the vertical 
axis (left-right head turning movements), head rota- 
tions about the lateral axis (up-down head move- 
ments), or combinations thereof. The foregoing head 
movements define the spatial perspective of the HMD 
subsystem, i.e., the pointing angles of the optical axis 
of the pilot's visual system (which may be defined by 
the radius vector of the polar coordinate system), 
which may be described in terms of the polar coordin- 
ate system. 

Transiational movements may be accommodat- 
ed and identified utilizing the processing capability of 
the primary processing, interface, and control means 
inasmuch as such movements involve computations 
based upon relatively straightforward geometric rela- 
tionships. The initialized spatial location and perspec- 
tive of the HMD subsystem is defined in terms of the 
design eyepoint and the optical axis of the pilot's vis- 
ual system. The initialized spatial location of the HMD 
subsystem is defined as the design eyepoint, i.e., the 
origin of the polar coordinate system. The initialized 
spatial perspective of the HMD subsystem is defined 
so that the optical axis of the pilot's visual system is 
coaxial with the longitudinal axis passing through the 
design eyepoint and medially through the forward por- 
tion of the cockpit canopy. The helmet sensing means 
is initialized to the foregoing initialized spatial location 
and perspective of the HMD subsystem. Subsequent 
movements of the HMD subsystem are detected by 
the helmet sensing means, and defined and descri- 
bed in terms of the polar coordinate system utilizing 
the processing capability of the primary processing, 
interface, and control means. 

The canopy structure typically includes a number 
of windshield posts, sideposts, and cockpit rails. Re- 
spective combinations of the windshield posts, side- 
posts, and cockpit rails define the viewing sectors of 
the canopy structure available to the pilot for visual 
observation of the external world. The windshield 
posts, sideposts, and cockpit rails occlude the pilot's 
visual observation of the external world during visual 
flight operations. As such, the pilot, either consciously 
or subliminally, references visual observations of the 
external world to such structural members as are in- 
terposed between the pilot and the external world. 
These structural members of the canopy structure are 
an integral element of the cognitive mechanism that 
allows the pilot to maintain spatial and situational 
awareness of the external world vis-a-vis the helicop- 
ter during visual flight operations. 

The VID system of the present invention incorpor- 
ates these structural members of the canopy structure 
into the virtual images that define the realistic, high- 
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definition pictorial representation provided to the pilot 
during non-visual flight operations (via the FLIR sub- 
system and the HMD subsystem) so that the pilot may 
rely upon and utilize the same cognitive mechanism 
that is operative during visual flight operations during 
non- visual flight operations to maintain spatial and sit- 
uational awareness of the external world vis-a-vis the 
helicopter. The digitized map is an interrelated series 
of structural outlines that correspond to the windshield 
posts, sideposts, and cockpit rails of the canopy struc- 
ture. Structural outlines corresponding to the wind- 
shield posts, the sideposts, and the cockpit rails are 
generated by means of the polar coordinate system 
and stored in the digitized map. 

The map is assembled in digitized form to facili- 
tate storage of the map in a memory structure in the 
primary processing, interface, and control means. 
During reconstruction of the relevant structural out- 
lines of the digitized map by the primary processing, 
interface, and control means for subsequent superim- 
positioning onto corresponding video images, the pri- 
mary processing, interface, and control means is op- 
erative to interpolate between adjacent digital points 
defining the respective structural outlines so that the 
reconstructed structural outlines are provided as con- 
tinuous structures. 

The primary processing, interface, and control 
(PPIC) means of the VID system is a general purpose 
computer that includes a central processing unit, 
memory storage, input/output ports, and a control, ad- 
dress, data bus. The PPIC means is operative to pro- 
vide electronic interfacing among the various subsys- 
tems comprising the VID system. 

The PPIC means is operative to provide control 
signals to synchronize the spatial orientation of the 
virtual imaging subsystem with the perspective of the 
HMD subsystem, i.e., to align the FOV of the virtual 
imaging subsystem in correspondence to the optical 
axis of the pilot's visual system. These control signals 
are derived by the central processing unit based upon 
the position/orientation signals provided by the hel- 
met sensing means. 

The PPIC means is further operative to provide 
the necessary processing capability required by the 
VID system of the present invention. This processing 
capability includes processing the virtual image sig- 
nals generated by the virtual imaging subsystem and 
providing corresponding signals to the HMD subsys- 
tem to generate corresponding video images. The 
PPIC means is also operative to define the spatial ori- 
entation and perspective of the HMD subsystem 
based upon position/orientation signals provided by 
the helmet sensing means. 

The PPIC means is also operative to correlate the 
defined spatial orientation and perspective of the 
HMD subsystem to the canopy structure of the heli- 
copter. Based upon the correlated spatial orientation 
and perspective of the HMD subsystem, the central 



processing unit reconstructs structural outlines of the 
relevant portion(s) of the canopy structure, utilizing 
the digitized map, to provide structural outlines that 
are synchronized with the virtual images generated by 
5 the virtual imaging subsystem. The central process- 
ing unit may be utilized to convert the synchronized 
structural outlines to a video format that is compatible 
with the video image generated by the HMD subsys- 
tem. 

10 The PPIC means is further operative to combine 

the synchronized structural outlines onto correspond- 
ing video images in a predetermined manner. More 
specifically, the synchronized structural outlines are 
superimposed onto corresponding video images such 

15 that the structural outlines occlude corresponding 
portions of the underlying video image. However, due 
to the configuration (thickness) of the structural out- 
lines, occlusion of the video images is minimized. 
The combined video images present a pictorial 

20 representation of the external world that is viewed by 
the pilot by means of the HMD subsystem and utilized 
by the pilot to maintain spatial and situational aware- 
ness of the external world vis-a-vis the helicopter 
based upon the cognitive mechanism. The combined 

25 video image display presents elements of the external 
world that are within the FOV of the virtual imaging 
subsystem. Correlating the structural outlines of the 
video images of the external-world provides the pilot 
with a spatial and situational awareness of the exter- 

30 nal world vis-a-vis the helicopter. 

Brief Description of the Drawings 

A more complete understanding of the present in- 
35 vention and the attendant features and advantages 
thereof may be had by reference to the following de- 
tailed description of the invention when considered in 
conjunction with the accompanying drawings where- 
in: 

40 Figure 1 is a schematic diagram of a virtual image 

display system incorporating structural outlines ac- 
cording to the present invention. 

Figure 2 is a plan view of a helicopter in which the 
virtual image display system of the present invention 
45 may be used. 

Figure 3 is a plan view of a helmet mounted dis- 
play subsystem having utility in the virtual image dis- 
play system of the present invention. 

Figures 4A, 4B illustrate a polar coordinate sys- 
so tern as a reference system for the virtual image dis- 
play system of the present invention. 

Figure 5A is a rectilinear plot of the canopy struc- 
ture of the helicopter of Figure 2. 

Figure 5B is a rectilinear plot illustrating a digi- 
55 tized map representing the optically opaque structural 
outlines corresponding to the canopy structure of Fig- 
ure 5A. 

Figure 6 is a schematic view of a primary process- 
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ing, interface, and control means for the virtual image 
display system of Figure 1. 

Figure 7 is a plan view of optically opaque struc- 
tural outlines superimposed onto a video display of a 
virtual image of the external world. 5 

Detailed Description of Preferred embodiments 

Referring now to the drawings where like refer- 
ence numerals identify corresponding or similar ele- 10 
ments throughout the several views, Figure 1 is a di- 
agrammatic illustration of a virtual image display (VI D) 
system 10 according to the present invention for use 
in piloting aircraft, particularly helicopters, during non- 
visual flight conditions such as night and/or adverse 15 
weather flight operations. The VID system 10 accord- 
ing to the present invention includes a primary proc- 
essing, interface, and control means 100, a virtual im- 
aging subsystem 20, a digitized map 80 of the aircraft 
canopy structure 60, a video display subsystem 40, 20 
and a means 50 for sensing the attitude and position 
of the video display subsystem 40 with respect to a 
predetermined aircraft coordinate system. 

The VID system 10 according to the present in- 
vention is operative to provide continuous virtual im- 25 
ages of the external world, to generate continuous 
video displays based upon these virtual images of the 
external world, and to transmit such video images to 
the pilot via the video display subsystem 40 for use in 
piloting the aircraft during non-visual flight conditions 30 
such as night and/or adverse weather flight opera- 
tions. The VID system 1 0 is further operative to super- 
impose a synchronized portion of the digitized map 80 
of the canopy structure 60 upon the video images 
such that the pilot is provided with a cognitive video 35 
display orientated with respect to the helicopter can- 
opy structure 60, thereby allowing the pilot to refer- 
ence the video displays representative of the external 
world vis-a-vis corresponding aircraft canopy struc- 
ture 60. 40 

As discussed hereinabove, a pilot either con- 
sciously or subliminally references visual images of 
the external world perceived by means of his visual 
system with corresponding parts of the aircraft cano- 
py structure during visual flight operations. This cog- 45 
nitive mechanism allows the pilot to maintain spatial 
and situational awareness of the external world vis-a- 
vis the aircraft The VID system 10 of the present in- 
vention is thus operative to provide the pilot with a 
cockpit referenced view of the external world during so 
non-visual flight conditions that is functionally and 
structurally equivalent to the pilot's view of the exter- 
nal world during visual flight conditions such that the 
cognitive mechanism that operates during visual flight 
conditions may be utilized by the pilot during non-vis- 55 
ual flight conditions to maintain spatial and situational 
awareness of the external world vis-a-vis the aircraft. 
The VID system 10 of the present invention precludes 



pilot disorientation during non-visual flight conditions, 
especially during flight conditions wherein the corre- 
sponding aircraft canopy structure 60 cannot be per- 
ceived by the pilot using his visual system. 

The VID system 10 of the present invention has 
particular utility for use in helicopters during non-vis- 
ual flight conditions such as night and/or adverse 
weather flight operations. One such helicopter H is ex- 
emplarily illustrated in Figure 2, which depicts the 
RAH-66 Comanche light helicopter. The RAH-66 hel- 
icopter H includes a transparent nose section NS that 
houses a gimballed virtual imaging subsystem 20 and 
a cockpit CP having a predetermined canopy struc- 
ture 60, as described in further detail hereinbelow. 
While the VID system 10 of the present invention is 
described hereinbelow in terms of the structure and 
configuration of the RAH-66 helicopter, it is to be un- 
derstood that the VID system 10 of the present inven- 
tion may be utilized in other helicopters and/or aircraft 
for non-visual flight operations. 

The virtual imaging subsystem 20 is operative to 
provide continuous virtual images of the external 
world during non-visual flight conditions, i.e., those 
flight conditions wherein the pilot's visual system is 
unable to comprehend the external world with suffi- 
cient clarity to ensure safe and/or efficient operation 
of the helicopter H, e.g., during night and/or adverse 
weather flight operations. The term virtual images as 
used herein defines a realistic, high-definition pictorial 
representation of the external world that corresponds 
in structural definition to the same external world if 
viewed by the human visual system, e.g., a tree or 
building is recognizable as a tree or building. 

The operating characteristics and capabilities of 
virtual imaging subsystems 20 of the type having util- 
ity in the VID system 10 of the present invention are 
well known to those skilled in the art For example, to 
generate virtual images of the external world during 
night flight operations, a forward looking infrared 
(FLIR) imaging subsystem 20 is the preferred means, 
based upon the state of present day technology, of 
generating continuous, high-definition virtual images 
of the external world during night flight operations. 
The FLIR imaging subsystem 20 senses differences 
in thermal radiation emitted by the various elements 
of the external world within the field of view (FOV) of 
the FLIR sensors and generates signals representa- 
tive of detected thermal radiation differences that are 
subsequently processed (see disclosure hereinbelow 
re the primary processing, interface, and control 
means 100) to provide continuous, high-definition vir- 
tual images of the external world within the FOV of the 
FLIR imaging subsystem 20. The FLIR imaging sub- 
system 20 may be selectively operated to generate 
hot and cold thermally differentiated images as black 
and white contrasts or vice versa, depending upon 
such factors as flight conditions, mission require- 
ments, etc. 
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Other imaging systems such low-light television 
systems, radar systems, or other sensors that are ca- 
pable of generating high-definition virtual images of 
the external world may be utilized as the virtual imag- 
ing subsystem 20 in the VID system 10 according to 
the present invention. As discussed hereinbelow in 
further detail, the virtual imaging subsystem 20 is 
slaved to the video image display subsystem 40 such 
that the FOV of the virtual imaging subsystem 20 is 
movably gimballed within the transparent nose struc- 
ture NS of the helicopter H to align the FOV of the 
FLIR subsystem 20 to coincide with the optical axis of 
the visual system of the pilot of the helicopter H. 

The video image display subsystem 40 of the VID 
system 1 0 is operative to position video images of the 
external world (derived from the virtual images cap- 
tured by the virtual imaging subsystem 20) centered 
with respect to the optical axis of the visual system of 
the pilot, i.e., within the field of view of the pilot With 
reference to Figure 3, the preferred video image dis- 
play subsystem 40 is a helmet mounted display 
(HMD) subsystem that includes a helmet 42 and a vi- 
sor type screen 44. The screen 44 may be fabricated 
as a reflective part of the optical train by utilizing a 
screen material that is partially reflective so that the 
screen 44 is operative to focus and project collimated 
video images onto the visual system of the pilot. Con- 
comitantly, the screen 44 may be fabricated to be par- 
tially transparent to visible light radiation from the ex- 
ternal world. The screen 44 is thus simultaneously op- 
erative to facilitate viewing of the external world (and 
the canopy structure 60) via the pilot's visual system 
during visual flight conditions and to project collimated 
video images, derived from virtual images provided by 
the virtual imaging subsystem 20, onto the pilot's vis- 
ual system that provide a realistic, high-definition pic- 
torial representation of the external world during non- 
visual flight conditions. The video images viewed by 
the pilot appear to be located at infinity due to the col- 
limation of the video images. 

The visor type screen 44 may be removably at- 
tached to the helmet 42 so that the screen 44 may be 
removed for visual flight operations if desired. The 
screen 44 may be fabricated with a defined curved 
configuration to provide a 30° X 60° degree panoram- 
ic field of view of the external world to the pilot (verti- 
cally and laterally, respectively, with respect to the 
optical axis of the pilot's visual system which is de- 
fined as a straight line projecting outwardly from the 
pilot's head from a point midway between his eyes). 

The HMD subsystem 40 may be binocular and 
may include dual miniaturized cathode ray tubes 
(CRTs) 46 such as those manufactured by Thomas 
Electronics, Hughes Aircraft, or AT&T Bell Laborator- 
ies, that receive image signals from the primary proc- 
essing, interface, and control means 100 and which 
are operative to generate high-definition video im- 
ages therefrom. A one-inch diameter CRT 46 may 



provide 1200 video lines, i.e., a 1200 X 1200 pixel ma- 
trix. Associated with each CRT 46 is a collimating opt- 
ical train 48 that provides the optical interface be- 
tween the CRTs 46 and the screen 44 for the high- 

5 definition video images, the optical train 48 being op- 
erative to magnify and collimate the video images 
generated by the CRTs 46 for projection of video im- 
ages onto the screen 44 for viewing by the pilot's vis- 
ual system. The HMD subsystem 40 may be operative 

10 to generate overlapping video images such that the 
pilot perceives a unitary, panoramic, high-definition 
video pictorial representation of the external world via 
the screen 44. HMD subsystems 40 having utility in 
the VID system 10 of the present invention are gen- 
ts erally known to those skilled in the art. A representa- 
tive example of a HMD subsystem is described in U.S. 
Patent No. Re 28,847. 

Associated with the helmet 42 of the HMD sub- 
system 40 is a sensing means 50 that is operative to 

20 define the spatial position and angular orientation of 
the HMD subsystem 40 within the cockpit CP of the 
helicopter H. As exemplarily illustrated in Figure 3, the 
helmet sensing means 50 comprises a sensor 52 
mounted in combination with the helmet 42 and an 

25 electromagnetic radiator 54 mounted in the cockpit 
CK adjacent the helmet 42. The sensor 52 is respon- 
sive to constant field strength electromagnetic radia- 
tion emitted by the electromagnetic radiator 54 to gen- 
erate position/orientation signals, corresponding to 

30 the position and angular orientation of the HMD sub- 
system 40 in the cockpit CK, based upon the field 
strength and phase of the intercepted electromagnet- 
ic radiation. These position/orientation signals are 
coupled to the primary processing, interface, and con- 

35 trol means 1 00 which is operative to process such sig- 
nals to identify the position and angular orientation 
(perspective) of HMD subsystem 40 in relation to the 
cockpit CK. Helmet sensing means 50 having utility in 
the VID system 10 of the present invention are gen- 

40 erally known to those skilled in the art, such sensing 
means being exemplarily illustrated by the three-axis 
Polhemus systems that provide signals indicative of 
the angular orientation of the helmet 42 in three axes 
as described in U.S. Patent Nos. 4,017,858 and 

45 3,983,474. 

An imaginary reference point 90, designated as 
the "design eye point", is defined in the cockpit CK of 
the helicopter H as illustrated in Figure 2 (imaginary 
in sense that the design eyepoint 90 does not coincide 

so with any structural elements or equipment comprising 
the canopy structure 60). The design eyepoint 90 is a 
design parameter that defines the degree of visibility 
(unimpaired visual field of view of the external world) 
provided to the pilot of the helicopter H based upon 

55 the structural configuration of the canopy structure 60. 
By way of example, for military helicopters having a 
single/tandem cockpit configuration, MIL-STD-850B 
sets forth suggested visibility requirements for the 
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canopy structure. 

For flight operations, the pilot positions himself in 
the cockpit seat CS (see Figure 2) and adjusts the 
cockpit seat CS so that the origin of the optical axis of 
his visual system coincides with the design eyepoint 
90 as illustrated in Figure 3. To facilitate this align- 
ment process, the RAH-66 helicopter H includes a 
boresight reticle unit mounted on the cockpit dash- 
board (coincident with the medial plane of the cockpit) 
and positioned so that, when activated, collimated 
light is emitted that passes through the design eye- 
point 90. Only when the pilot is properly positioned in 
the cockpit CP, i.e., the origin of the optical axis of his 
visual system is coincident with the design eyepoint 
90, will this collimated light be perceptible to the pilot* s 
visual system. 

The design eyepoint 90 defines an origin (0, 0, 0) 
of a coordinate system for the VID system 10 of the 
present invention. To facilitate the identification and 
description of the spatial location and perspective of 
the HMD subsystem 40 by means of the primary proc- 
essing, interface, and control means 100, a polar co- 
ordinate system (r, ©, J3, a), as exemplarily illustrated 
in the plan views of Figures 4A, 4B, which are overlaid 
on the corresponding cockpit configuration of the hel- 
icopter H, is utilized to define the spatial location and 
perspective of the HMD subsystem 40 and to gener- 
ate the digitized map 80 of the canopy structure 60, 
as discussed in further detail hereinbelow. Also illu- 
strated in Figures 4A, 4B is the three axes coordinate 
system generally used for helicopter descriptive pur- 
poses, i.e., the longitudinal axis X-X, the transverse 
axis Y- Y, and the vertical axis Z-Z. As described here- 
inbelow in further detail, the polar coordinate system 
may be used to define any point (x, y, z) of the canopy 
structure 60 by means of the transformation equa- 
tions: 

x = r(cos0) 

y = r(cosp) 

z = r(cosa) 
r 2 = x 2 + y 2 + z 2 
Identifying the spatial location and perspective of 
the HMD subsystem 40 is generally simplified by use 
of the polar coordinate system. As a general rule, 
once a pilot is properly positioned in the cockpit seat 
CS, as described hereinabove, the majority of head 
movements (which directly correspond to HMD sub- 
system 40 movements) naturally made by the pilot 
during typical flight operations consist of head rota- 
tions about the vertical axis Z-Z (left-right head turning 
movements), head rotations about the lateral axis Y- 
Y (up-down head movements), or combinations 
thereof. The foregoing head movements define the 
spatial perspective of the HMD subsystem 40, i.e., the 
pointing angles of the optical axis of the pilots visual 
system (which may be defined by the radius vector of 
the polar coordinate system), which may be described 
in terms of the angle 0, the angle a, or combinations 



thereof, respectively, of the polar coordinate system. 

Experience has shown that there will be relatively 
little translational movement of the pilot's head along 
the longitudinal, lateral, and/or vertical axes during 

5 normal flight operations. Pragmatically, therefore, the 
HMD subsystem 40 may be defined as having a con- 
stant spatial location coincident with the origin of the 
polar coordinate system. Even if the HMD subsystem 
40 is subjected to translational movements, such 

10 movements may be accommodated and identified 
utilizing the processing capability of the primary proc- 
essing, interface, and control means 100 inasmuch as 
such movements involve computations based upon 
relatively straightforward geometric relationships. 

15 The initialized spatial location and perspective of 

the HMD subsystem 40 is defined in terms of the de- 
sign eyepoint 90 and the optical axis of the pilot's vis- 
ual system. The initialized spatial location of the HMD 
subsystem 40 is defined as the design eyepoint 90, 

20 i.e., the origin of the polar coordinate system. The ini- 
tialized spatial perspective of the HMD subsystem 40 
is defined so that the optical axis of the pilot's visual 
system is coaxial with the longitudinal axis X-X pass- 
ing through the design eyepoint 90 and medially 

25 through the forward portion of the cockpit CP canopy. 
The helmet sensing means 50 is initialized to the fore- 
going initialized spatial location and perspective of the 
HMD subsystem 40. Subsequent movements of the 
HMD subsystem 40 are detected by the helmet sens- 

30 ing means 50, and defined and described in terms of 
the direction angles 0, a (or the radius vector r and the 
direction angle p, as required) utilizing the processing 
capability of the primary processing, interface, and- 
control means 100. 

35 Figure 5A is a rectilinear plot of the canopy struc- 

ture 60 of the helicopter H illustrated in Figure 2. The 
rectilinear plot was derived based upon the design 
eyepoint 90 and the polar coordinate system descri- 
bed hereinabove. The abscissa represents the direc- 

40 tion angle 0 and the ordinate represents the direction 
angle a. The canopy structure 60 includes left and 
right windshield posts 61 L, 61 R, a transverse post 62, 
left and right forward sideposts 63L, 63 R, left and right 
rearsideposts 64L, 64R, left and right secondary side- 

45 posts 65L, 65R, a forward cockpit rail 66, left and right 
primary cockpit rails 67L, 67R, left and right second- 
ary cockpit rails 68L, 68R, and left and right tertiary 
cockpit rails 69L, 69R. 

The foregoing structural members comprising the 

so canopy structure 60 in respective combinations de- 
fine the viewing sectors of the canopy structure 60 
available to the pilot for visual observation of the ex- 
ternal world. These may be defined as the forward 
canopy 70, the left and right primary side canopies 

55 72L, 72R, the left and right secondary side canopies 
74L, 74R, and the left and right tertiary side canopies 
76L, 76R as illustrated in Figure 5A. The various 
canopies described above are fabricated from an opt- 
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ically transparent material such that visible light radi- 
ation from the external world is transmitted there- 
through to the visual system of the pilot The canopy 
members are not relevant to the VID system 10 of the 
present invention inasmuch as these members are ef- 
fectively transparent to the pilot's visual system, and 
as such, do not function as reference points for visual 
observations of the external world by the pilot during 
visual flight operations, and therefore, are not part of 
the pilot's cognitive mechanism for maintaining spa- 
tial and situational awareness of the external world 
vis-a-vis the helicopter H. 

In contrast, the structural posts and rail members 
of the canopy structure 60 occlude the pilot's visual 
observation of the external world during visual flight 
operations. As such, the pilot, either consciously or 
subliminally, references visual observations of the ex- 
ternal world to respective structural posts and rail 
members that are interposed between the pilot and 
the external world. The structural posts and rail mem- 
bers of the canopy structure 60 as described hereina- 
bove are an integral element of the cognitive mecha- 
nism that allows the pilot to maintain spatial and sit- 
uational awareness of the external world vis-a-vis the 
helicopter H during visual flight operations. 

Therefore, the VID system 10 of the present in- 
vention incorporates these elements of the canopy 
structure 60 into the virtual images that define the re- 
alistic, high-definition pictorial representation provid- 
ed to the pilot during non-visual flight operations (via 
the FLIR subsystem 20 and the HMD subsystem 40) 
so that the pilot may rely upon and utilize the same 
cognitive mechanism that is operative during visual 
flight operations during non-visual flight operations to 
maintain spatial and situational awareness of the ex- 
ternal world vis-a-vis the helicopter H. 

Figure 5B is a to-scale rectilinear digitized map 80 
corresponding to the canopy structure 60 illustrated in 
Figure 5B and described hereinabove. The digitized 
map 80 is an interrelated series of structural outlines 
that correspond to the structural posts and rails of the 
canopy structure 60 as described hereinabove. Struc- 
tural outlines 81 L, 81 R correspond to the left and right 
windshield posts 61 L, 61 R, respectively, structural 
outline 82 corresponds to the transverse post 62, 
structural outlines 83L, 83R correspond to the left and 
right forward sideposts 63 L, 63R, respectively, struc- 
tural outlines 84 L, 84R correspond to the left and right 
rear sideposts 64 L, 64R, respectively, structural out- 
lines 85L, 85R correspond to the left and right second- 
ary sideposts 65L, 65R, respectively, structural out- 
line 86 corresponds to the forward cockpit rail 66, 
structural outlines 87L, 87R correspond to the left and 
right primary cockpit rails 67L, 67R, respectively, 
structural outlines 88L, 88R correspond to the left and 
right secondary cockpit rails 68L, 68R, respectively, 
and structural outlines 89L, 89R correspond to the left 
and right tertiary cockpit rails 69L, 69R. 



The digitized map 80 may be formed by utilizing 
the polar coordinate system described hereinabove to 
define digitized structural outline sequences (based 
upon digital points defined and identified by the polar 

5 coordinate system) corresponding to each of the 
posts and rails comprising the canopy structure 60. 
The map 80 is assembled in digitized form to facilitate 
storage of the map 80 in a memory structure in the pri- 
mary processing, interface, and control means 100. 

10 During reconstruction of the relevant structural out- 
lines of the digitized map 80 by the primary process- 
ing, interface, and control means 100 for subsequent 
superimpositioning onto corresponding video images, 
the primary processing, interface, and control means 

15 100 is operative to interpolate between adjacent dig- 
ital points defining the respective structural outlines 
so that the reconstructed structural outlines are pro- 
vided as continuous structures. The primary process- 
ing, interface, and control means 100 may be further 

20 operative to vary the thickness of the structural out- 
lines, as required. 

The primary processing, interface, and control 
(PPIC) means 100 of the VID system 10 is exemplarily 
illustrated in further detail in Figure 6. The PPIC 

25 means 100 exemplarily illustrated includes a central 
processing unit 1 02, a random access memory (RAM) 
103, a read only memory (ROM) 104, input/output 
ports 106, 108, 110, 1 12, and a control, address, data 
bus 114. The PPIC means 100 is illustrated in gener- 

30 alized form as a general purpose computer that may 
take on forms different than that specifically illustrat- 
ed. 

The computer architecture may be of the split 
processor type having more than one control, ad- 

35 dress, data bus. One bus may be dedicated to in- 
put/output tasks and communication functions. An- 
other separate bus may be dedicated to image proc- 
essing, definition functions, and map line reconstruc- 
tion and conversion to video format. For architecture 

40 employing more than a single bus, each bus would 
typically have a processor such as a Motorola 80286 
type processor associated therewith. The PPIC 
means 100 may be implemented in the form of Very 
High-Speed Integrated Circuitry (VHSIC) with 1.25 

45 microns between geometric features. Such a PPIC 
means 100 may have more than 1 3 megabytes of in- 
ternal memory, and the capability of executing 1 9 mil- 
lion instructions per second and 300+ million opera- 
tions per second. 

50 The PPIC means 1 00 is operative to provide elec- 

tronic interfacing among the various subsystems 
comprising the VID system 10 as described hereina- 
bove, via signal lines 1 1 6, 1 1 8, 1 20. 1 22, 1 24, respec- 
tively. Interactive coupling between the HMD subsys- 

55 tern 40 and the helmet sensing means 50 is identified 
by reference numeral 130. 

The PPIC means 100 is operative to provide con- 
trol signals via signal line 1 16 to synchronize the spa- 
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tial orientation of the virtual imaging subsystem 20 
with the perspective of the HMD subsystem 40, i.e., 
to align the FOV of the virtual imaging subsystem 20 
in correspondence to the optical axis of the pilot's vis- 
ual system. These control signals are derived by cen- 5 
tral processing unit 102 based upon the position/ori- 
entation signals provided by the helmet sensing 
means 50 via signal line 122. 

The PPIC means 100 is further operative to pro- 
vide the necessary processing capability required by 1 o 
the VID system 1 0 of the present invention. This proc- 
essing capability includes processing the virtual im- 
age signals generated by the virtual imaging subsys- 
tem 20 and inputted via signal line 188 to provide cor- 
responding signals to the HMD subsystem 40, via sig- 15 
nal line 120, for generation of video images corre- 
sponding to the virtual image. The PPIC means 100 
is also operative to define the spatial orientation and 
perspective of the HMD subsystem 40 based upon 
position/orientation signals provided by the helmet 20 
sensing means 50 via signal line 122. 

The PPIC means 100 is also operative to corre- 
late the defined spatial orientation and perspective of 
the HMD subsystem 40 to the canopy structure 60 of 
the helicopter H by means of the central processing 25 
unit 102. Based upon the correlated spatial orienta- 
tion and perspective of the HMD subsystem 20, the 
central processing unit 102 reconstructs structural 
outlines of the relevant portion(s) of the canopy struc- 
ture 60, utilizing the digitized map 80, to provide struc- 30 
tural outlines that are synchronized with the virtual im- 
ages generated by the virtual imaging subsystem 20. 
The central processing unit 102 may be utilized, to 
convert the synchronized structural outlines to a video 
format that is compatible with the video image gener- 35 
ated by the HMD subsystem. 

The PPIC means 100 is further operative to com- 
bine the synchronized structural outlines onto corre- 
sponding video images in a predetermined manner 
via signal line 120. More specifically, the synchron- 40 
ized structural outlines are superimposed onto corre- 
sponding video images such that the structural out- 
lines occlude corresponding portions of the underly- 
ing video image. However, due to the configuration 
(thickness) of the structural outlines, occlusion of the 45 
video images is minimized. 

Figure 7 is a pictorial representation of an instan- 
taneous combined video image generated by means 
of the VID system 10 of the present invention. This 
pictorial representation is viewed by the pilot by so 
means of the HMD subsystem 40 and utilized by the 
pilot to maintain spatial and situational awareness of 
the external world vis-a-vis the helicopter H based 
upon the cognitive mechanism described hereina- 
bove. The pictorial representation of Figure 7 illus- 55 
trates structural outlines 81 L, 81 R, corresponding to 
the left and right windshield posts 61L, 61 R of the hel- 
icopter H, superimposed upon a video image gener- 



ated by the HMD subsystem 40 which was derived 
from a virtual image provided by the virtual imaging 
subsystem 20. The video image displays elements of 
the external world such as a helicopter He in flight, 
trees T, and a building B that are within the FOV of the 
virtual imaging subsystem 20. Other symbology illu- 
strated in Figure 7 is representative of aircraft status 
information that is not pertinent to the present inven- 
tion. The symbol identified by the reference letters 
AHL is an artificial horizon line corresponding to the 
horizon line of the external world. 

With respect to Figure 7, the structural outlines 
81 L, 81 R apprise the pilot that the displayed pictorial 
representation is being viewed through the forward 
canopy 70 of the helicopter H. Correlating the struc- 
tural outlines 81 L, 81 R with the artificial horizon line 
AHL apprises the pilot that the helicopter H is being 
flown in a shallow bank to the left This information 
provides the pilot with spatial awareness of the rela- 
tionship between the helicopter H and the external 
world utilizing the cognitive mechanism described 
hereinabove. 

Correlating the structural outlines 81 L, 81 R with 
the pictorial display of the helicopter He apprises the 
pilot that the helicopter He may be on a flight path that 
will intersect the pilot's course. The trees T and build- 
ing B apprise the pilot of the proximity of the helicopter 
H that may present hazards to forward flight. This in- 
formation provides the pilot with situational aware- 
ness of the relationship between the helicopter H and 
the external world utilizing the cognitive mechanism. 

A variety of modifications and variations of the 
present invention are possible in light of the above 
teachings. It is therefore to be understood that, within 
the scope of the appended claims, the present inven- 
tion may be practiced otherwise than as specifically 
described hereinabove. 



Claims 

1. A virtual image display system for an aircraft hav- 
ing a canopy structure of predetermined configur- 
ation, comprising: 

virtual imaging means for generating vir- 
tual images of the external world; 

video display means for generating video 
images based upon said virtual images of the ex- 
ternal world for viewing by the aircraft pilot; 

sensing means disposed in interactive re- 
lation with said video display means for generat- 
ing signals corresponding to the spatial location 
and perspective of said video display means; 

map means for storing structural outlines 
corresponding to the predetermined configura- 
tion of the canopy structure; and 

computer processing means for defining 
the spatial location and perspective of said video 
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display means based upon said signals generat- 
ed by said sensing means, 

said computer processing means being 
further operative for reconstructing structural out- 
lines from said map means based upon and syn- 5 
chronized in perspective with said defined spatial 
location and perspective of said video display 
means, and wherein 

said computer processing means is further 
operative to superimpose said synchronized 10 
structural outlines onto said video images to pro- 
vide the pilot with a high-definition pictorial repre- 
sentation of the external world wherein said su- 
perimposed, synchronized structural outlines 
provide the pilot with a referential framework. 15 
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